Validation of average error rate over classifiers

We examine methods to estimate the average and variance of test error rates over a set of classi ers We begin with the process of drawing a classi er at random for each example Given validation data the average test error rate can be estimated as if validating a single classi er Given the test example inputs the variance can be computed exactly Next we consider the process of drawing a classi er at random and using it on all examples Once again the expected test error rate can be validated as if validating a single classi er However the variance must be estimated by validating all classifers which yields loose or uncertain bounds